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This report describes West Virginia's Basic Skills/Computer 
Education (BS/CE) program and connects its features to gains in student test 
scores that are practically and statistically significant. The program 
consists of three components: (1) software that focuses on the state's basic 

skills goals in reading, language arts, and mathematics; (2) enough computers 
in the schools so that all students will be able to have easy and regular 
access to the basic skills software; and (3) professional development for 
teachers in the use of the software and the use of computers in general. Data 
were collected from all fifth graders (n=950) in 18 elementary schools 
selected to represent the range of variables that might influence technology 
use and student achievement. Survey data were also collected from 290 
teachers. Results were analyzed using a model that includes access to 
software and computers, attitudes toward technology, and teacher training and 
involvement. The relationship of BS/CE to student achievement, as measured by 
the Stanford- 9 achievement test, is examined. Findings suggest that the BS/CE 
program had a powerfully positive effect, especially in those schools that 
used it most intensively. Ten figures and tables illustrate findings. (AEF) 
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Under what conditions is technology effective in advancing learning and teaching is the fundamental question that 
drives the research interests of the Milken Exchange on Education Technology. The West Virginia Basic Skills/Computer 
Education (BS/CE) technology implementation beginning nearly ten years ago deserves our scrutiny because of its 
scale, consistency and focus. What can we learn from an approach that spanned a whole state for a full decade? The 
issues of system design, training, technology capacity, technical support, and means of measurement are all 
powerfully present in the West Virginia experience. The goal of the Exchange in commissioning the study was not to 
praise it or criticize it, but to understand it and to make that understanding known to others. 

The findings of this study suggest strongly that the BS/CE program had a powerfully positive effect in West 
Virginia, especially in those schools that used it most intensively. 

• Significant gains in reading, writing, and math were achieved. 

• BS/CE was found to be more cost-effective than other popular interventions including class-size reduction. 

• The program was especially successful with low income and rural students as well as with girls. 

But these findings need to be interpreted cautiously by educators and policymakers because: 

• BS/CE is based on instructional learning systems designed over a decade ago and limited to the then 
available technology. For example, easy access to the Internet was just a dream in 1989. 

• The pedagogy upon which instructional learning systems are based makes little use of project-based 
learning and other constructivist curricular approaches that are the leading edge of learning technology today. 

• BS/CE fit the learning and teaching realities of West Virginia over the last decade. That does not make it 
appropriate for every district or state where the characteristics of learners and teachers may be quite different. 

Learning technology evolves quickly because of changes in technology and advances in our understanding of how to 
use it to engage, challenge and nurture learners. The lag between the introduction of a program and evidence of its 
effectiveness may be significant. Still, it is useful to know what works and in what ways. The future forms of 
learning technology are impossible to predict, but we can design them better based on the islands of research that 
help explain where we have been. 



Cheryl Lemke 
Executive Director 

Milken Exchange on Education Technology 
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The West Virginia Department of Education is pleased to collaborate with the Milken Family Foundation in the 
publication of a landmark study documenting the powerful and positive impact of the Basic Skills/Computer 
Education program on student achievement in West Virginia. 

Our experience and data tell us that the Basic Skills/Computer Education program is successful. West Virginia has had 
across-the-board increases in statewide assessment scores in all basic skills areas, and our NAEP scores have improved. 
Just as importantly, we have seen the faces of children light with excitement and learning and have heard the 
renewed enthusiasm of teachers when technology is integrated into the curriculum. 

The reasons for West Virginia's success are numerous. Clearly articulated goals focus upon increased student achieve- 
ment in reading, mathematics, and composition. Software was aligned with the West Virginia Instructional Goals and 
Objectives. Implementation, which began in the earliest grades, moves upward each year. Comprehensive and 
timely staff development enables teachers to correlate the software with the curriculum and integrate the technology 
into instruction. The state-purchased computer systems are distributed equitably and provide a consistent 
platform for instructional and support purposes. Administrative support at the county, region, and state board of 
education levels provides clear evidence of program importance. All of these factors were developed through partic- 
ipating stakeholders who provided a bottom-up approach that created a statewide solution for technology integration. 

West Virginia's model calls for increasingly higher standards for student achievement. As our model continues to be 
buttressed by implementation of best practices identified in solid research. West Virginia students will progress to 
meet these higher standards. This study adds to the limited body of solid research on the impact of learning tech- 
nologies on student achievement. 

The Milken Family Foundation is to be commended for promoting research aimed at identifying those factors that 
contribute to student achievement with the use of learning technologies. This study is a collaboration among the 
West Virginia Department of Education, the educators and students in the schools studied, the Milken Family 
Foundation, and Interactive, Inc. This collaborative effort is truly appreciated. Increasing student achievement remains 
our central focus, and the effective implementation of technology will continue to advance this goal. 



Henry R. Marockie 

State Superintendent of Schools 

West Virginia 
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WEST VIRGIIMIA STORY: 
Achievement gains from 
a statewide comprehensive 
instructionai technaiogy program 



EXECUTIVE SUMMARY 



The Program. 

West Virginias "Basic Skills/Computer Education" program is unique in its eight year longevity and in its 
documented student achievement outcomes. This report describes the program and connects its features to 
gains in student test scores that are practically and statistically significant. 

The Basic Skills/Computer Education (BS/CE) program was authorized in 1989-90 and, beginning with the 
kindergarten class of 1990-91, hardware and software were installed in schools and teacher training began. 
The program consists of three basic components: 

(1) software that focuses on the State's basic skills goals in reading, language arts and mathematics; 

(2) enough computers in the schools so that all students will be able to have easy and regular access 
to the basic skills software; and 

(3) professional development for teachers in the use of the software and the use of computers in general. 

Each year from 1990-91 onward and beginning with kindergarten, at a cost of about $7 million per year, the 
State of West Virginia provided every elementary school with enough equipment so that each classroom 
serving the grade cohort of children targeted that year might have three or four computers, a printer and a 
school-wide, networked file server. Schools could choose to deploy the computers in labs and centers or dis- 
tribute them directly to classrooms. As the 1990-91 kindergarten class went up the grades, so did the 
successive waves of new computer installations coupled with intensive professional development for 
teachers and software chosen from either IBM or Jostens Learning. 



Research Methods. 

We collected data from all fifth graders (950) in 18 elementary schools that were selected to represent the 
range of variables that might influence technology use and student achievement, e.g., intensivity of BS/CE 
use, software vendor, student prior achievement and sociodemography. The 1996-97 fifth graders had the 
most complete test score records and they were the first cohort to have had the consistent availability of 
BS/CE across their entire school experience. The sample size supports generalization at the 95% level of 
statistical confidence. We collected survey data from 290 teachers in the study schools. Data were both 
quantitative (state and publisher's test files, survey results) and qualitative (on-site field documentation, case 
analysis, interview results). 
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Student test data were scaled scores on the Stanford-9 achievement test. Because scaled scores are 
normed against a nationally representative group, they are appropriate for comparison purposes and for the 
computation of gain scores. Our emphasis is on gains in "basic skills," a score computed and reported by 
the West Virginia Department of Education for comparison purposes, and a score that measures the math, 
reading and language arts areas that were the focus of BS/CE. 

Explaining Test Score Variation: The Access/Attitude/Training Model. 

We used factor analysis to search for input phenomena that were grouped both conceptually and in terms of 
respondent perceptions and that were also related to variation in student test scores. The three components 
of this empirically-derived model — access, attitude, and training — are similar to what leaders in instructional 
technology advocate. Unless there is sufficient equipment and opportunity to learn (access) there are unlike- 
ly to be effects from instructional technology. And, unless teachers have a chance to learn about the 
technology and how it may make them and their students more successful (training), they are less likely to 
believe it can help (attitude) and, in fact, to use it. 

B5/CE Effect on Student Achievement. 

The more of each of the model components that the student experienced, the higher the gain score on the 
Stanford-9. The BS/CE technology regression model accounts for 1 1% of the total variance in the basic skills 
achievement gain scores of the 5th-grade students. That impact is powerful and practically significant; it is 
also statistically significant at more than the .001 level. More to the point, an 1 1 % improvement in test scores 
would be welcomed by most students, parents and educators. 

But, we believe that the 11% explained by our model underestimates the value added by instructional tech- 
nology. First, schools are only one of several educators. Family background explains a great deal about why 
some children do better than others. Since James Coleman's 1965 research, we have known that children's 
achievement in school is conditioned more by what they bring to school than by what schools are able to do 
with them. Consider: 

(D)espite the wide range of diversity of school facilities, curriculum and teachers, and despite the wide 
diversity among student bodies in different schools, over 70% of the variation in achievement for each 
group is variation within the same student body (James S. Coleman, et al. Equality of Educational 
Opportunity. Washington DC., US Department of Health, Education and Welfare, 1966, p. 77). 

If, as Coleman and others assert, 70% of the variation in test score performance relates to family and other 
home and background factors, that leaves 30% that schools can influence. We call that 30%, "school acces- 
sible performance." 

This analysis documents that as much as 11% of the gain score variance for one year can be explained by 
BS/CE. Thus, BS/CE explains more than a third of the "school" reasons why students' achievement scores 
improved. Said another way, of ail the factors that can affect a child's learning, about 30% is within the 
school's sphere of influence. The BS/CE technology initiative explained nearly 11% of that 30%. 



ERIC 
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There are two additional reasons to believe that the reported gain scores underestimate the total effect of 
BS/CE. The 11% represents gain score variation explained by BS/CE within a single year, i.e., 1997 to 1998. 
However, these fifth graders had BS/CE for four previous years. Thus, the cumulative effect of BS/CE most 
likely accounts for more than the 1 1% of the single year gains that we found. 

We also suspect that the gain scores reported are unnecessarily conservative because current hardware and 
software is more powerful than what was implemented in BS/CE's early years. Current technology is likely 
to yield even larger gains. 

The Distribution of Achievement Gains Across Groups of Children. 

BS/CE helped all children perform better, but the data Indicate that BS/CE helped the neediest children the 
most. Those children without computers at home made the biggest gains in (1) total basic skills, (2) total 
language, (3) language expression, (4) total reading, (5) reading comprehension, and (6) vocabulary. Also, 
the Access/ Attitude/Training model explains more of the basic skills gain scores for students who report 
lower grades. 

Although the relative disadvantage of girls is a regularity of the technology and the gender literatures, girls 
and boys reported the same access and the same use of computers in West Virginia. The more years that 
girls report having used computers, the more they like them and the more they report knowing about them. 
(Students reported increased computer use every year from kindergarten through the fourth grade.) Girls 
reported that computers were more accessible to their particular learning needs than were their teachers. In 
math and reading outcomes, there were no gender differences. 

Teachers and B5/CE. 

More than half the State's teachers are confident in using computers in their teaching, only 1 9% are not. Half 
the teachers thought that technology had helped "a lot with West Virginia instructional goals and objectives." 
Almost half the teachers became more enthusiastic about BS/CE as time passed. 



Implementing B5/CE. 

BS/CE was fielded as it was designed — with a critical mass of hardware, software and training tightly focused 
on a grade-by-grade follow through schedule and on basic skills acquisition. 

Schools could choose how to deploy the BS/CE computers — concentrated in labs, distributed to classrooms 
or a combination of lab-plus-classroom distributed. The deployment choice reflects compliance with the cur- 
rent goal of "technology integration into classrooms,” and is thus significant for policy and crucial to teachers. 

Students who had access to BS/CE computers in their classrooms (the "distributed" pattern) did signifi- 
cantly better than students who were taught with BS/CE equipment in lab settings. They had higher gains 
in overall scores and in math. They also scored higher on the 1998 tests than did those in labs. (There were 
no differences on scores by source of software.) 
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Teachers who had computers in the classroom reported higher skill levels In delivering instruction, 
planning lessons, managing paperwork and word processing. Teachers who had BS/CE computers in 
their classrooms also reported more time using BS/CE computers for reading, math and writing instruction 
than either of the other two patterns. Sixty-one percent of the teachers with access to computers in their 
own classrooms said they were confident in using computers in their teaching compared to only 43 percent 
of the teachers who took their children to a lab for instruction in, about, or with computers. 

Additional Explanations for Test Score Gains. 

During the period studied, the State renovated 470 schools and built 68 new buildings. Between 1990 and 
1993, increases to the average teacher's salary moved West Virginia up from 49th to 34th among the states. 
Still 48% of the teachers chose "technology" as the number one explanation for student learning gains. 

The state also required or instituted "Unified School Improvement Planning," a statewide curriculum frame- 
work ("West Virginia Instructional Goals and Objectives"), local school councils, faculty senates, school site 
accreditation visits and a "probationary" procedure. Also during this period. West Virginia Bell connected 700 
of the State's 840 school buildings to Bell Atlantic's "World School" Internet service. The measured pres- 
ence of "other technology related initiatives" accounts for 0.4% of the variance in achievement scores. Each 
of those initiatives probably accounts for parts of the total improvement although it was beyond the scope of 
this analysis to establish those amounts. 

Conclusions. 

This analysis establishes how much value can be added on a statewide basis from a sustained instructional 
technology initiative. The data indicate that as much as a third of the gains in "school accessible achievement" 
can be powered by instructional technology. The data also signal aspects of policy strategy and tactics. 

BS/CE is scalable. The expenditure proportions are not beyond reach for other states. (See also Lewis Solmon's 
"Afterword.") The outcomes are established. The program's components have been well documented. 

As other Jurisdictions consider instructional technology as an agent of improvement, is it of interest that a 
package of hardware/software/process innovations can account for a large fraction of the test score improve- 
ment that is available to public policy intervention? And is it of further interest that, in addition to test score 
gains, those innovations can help position children for a technologically demanding economy, society and polity? 

BS/CE also has a number of features that are uncommon in the state education policy landscape. The 
features that deliver an installed base critical mass depart from the norm of a small number of computers 
equitably distributed across a large number of classrooms. The choice of software from a fixed set of two 
vendors departs from the conventional ceding of choice among hundreds of vendors to hundreds of schools 
(and often, to thousands of teachers). We believe that part of the explanation for BS/CE's success is the 
defined focus of its implementation. 

Still, policy choices, political choices always honor local values. In American schooling, thousands of Juris- 
dictions make their own choices. It may be that this documentation of the student outcomes associated with 
West Virginia's program of Basic Skills/Computer Education will advance the consideration of similar initia- 
tives in other Jurisdictions. 




WEST VIRGIIMIA STORY: 
Achievemen't gains fram 
a statewide comprehensive 
instructional technology program 



1.0 INTRODUCTION 

West Virginia's instructional technology initiative is unique in its eight year longevity and in its documented 
student achievement outcomes. This report describes the results of quantitative and qualitative analyses con- 
ducted with the support of the Milken Exchange on Education Technology across a stratified sample of 
schools in the state. 

1.1 West Virginia's "Basic Skills/Computer Education" (BS/CE) Program 

The Basic Skills/Computer Education (BS/CE) program was authorized in 1989-90 and, beginning with the 
kindergarten class of 1990-91, hardware and software were installed in schools and teacher training began. 
BS/CE was intended to improve the basic skills learning of West Virginia's elementary students through tech- 
nology. The program consists of three basic components: (1) software that focuses on the State's basic skills 
goals; (2) enough computers in the schools so that all students would have easy and regular access to the basic 
skills software; and (3) training for teachers in the use of the software and the use of computers in general. 

Each year from 1990-91 onward and beginning with kindergarten, the State of West Virginia provided every 
school with enough equipment so that each classroom serving the grade cohort of children targeted that year 
might have three or four computers, a printer and a school-wide, networked file server. Schools could choose 
to deploy the computers in labs and centers or distribute them directly to classrooms. As that kindergarten 
class went up the grades, so did the successive waves of new computer installations coupled with intensive 
professional development for teachers and software chosen from either IBM or Jostens Learning. The soft- 
ware and training emphasized the basic skills of reading, mathematics and writing. 

The BS/CE that students and teachers experienced was an obvious amount of new gear (either concentrat- 
ed in a showcase lab or center or distributed directly to classrooms); new software that related directly to a 
consistent, statewide priority on basic skills instruction^ and intensive training prior to implementation cou- 
pled with continuous support during the early implementation. All elementary teachers on a given grade level 
were experiencing the same software, the same expectations, the same new challenges, and the same 
opportunities. BS/CE thus covered all the bases of technology integration — hardware, software, and pro- 
fessional development and involvement — and all in a concentrated, sustained and visible program. 



’ This analysis was designed as a study of the aggregate impact of all of BS/CE’s features on educational achievement. It was not designed to measure the rela- 
tive merits of the two software vendors, IBM and Jostens. The software packages that each made available were targeted on basic skills instruction. To make the 
software useful for teachers and to relate it to state priorities, the vendors mapped their packages onto the State’s curriculum objectives. Finally, each package 
included Integrated Learning Systems features— student and class data reports to support individual, classroom, grade-level, and school -wide analysis and action. 



BS/CE is important because of how it was done — the inputs — and because of what happened — the 
outcomes for students, teachers and schools. We first describe the program. 

1.1 A Policy Inputs 

In program design, the grade-by-grade follow-through strategy is different from conventional practice. 
Because the State could target its expenditures in equipping a single year of classrooms, the concentration 
of computers potentially available per classroom was much higher than would be the case if the same num- 
ber of computers were distributed evenly, and thinly, across seven times as many classrooms (K-6). 

The investment in professional development also departs from the usual practice. West Virginia spent roughly 
30(t of every technology dollar on training, ten times the national average for schools. The State conforms to 
the recommendations of the U.S. Department of Education that 30 percent of the total technology budget be 
spent on professional development for teachers^. In one two-year period, 5,000 teachers were provided with 
professional development in a 'turn-key training' process provided by the State pursuant to a state contract 
with the software vendors. 

In contrast to the ordinary laissez-faire local selection of software, the state provided two sources of 
software — IBM or Jostens — between which local jurisdictions could choose. Jostens offered the Jostens 
Learning System in Reading and Mathematics, and IBM made available their Basic Skills Courseware. 
The Jostens Learning System provided identical software across schools; therefore, students who used the 
Jostens Learning System in one school were exposed to the same intervention as students in another school. 
The IBM offering, however, included a number of programs from which teachers could choose^. Each soft- 
ware package was developed or adapted by the vendor to emphasize the basic skills targeted by 
West Virginia for improvement — reading and mathematics'^. Each package was mapped onto West Virginia's 
statewide instructional goals and objectives. 

The option of two providers — IBM and Jostens Learning — allowed counties to select the solution that most 
closely aligned with the educational philosophy of the district. While both solutions meet the intent of the 
state program, they differ in their approach. Both solutions are strongly correlated to national standards and 
state instructional goals. Additionally, both were available to be implemented in classroom or lab configura- 
tions depending on the individual school's choice. 

IBM fosters a learning center approach in which students move in small groups to complete specific activi- 
ties at various classroom or lab locations designated as centers. Each session begins with a whole group 
activity and then moves to independent or cooperative use of the computer and courseware. According to 
IBM, this approach promotes problem-solving and higher-order thinking skills. The courseware offered by IBM 
addresses reading, mathematics and writing skills, while offering the districts some range of choices among 
software programs as teachers build a solution that complements and extends classroom instruction. 

^ US Department ot Education. Getting America’s Students Ready tor the 21st Century, Washington. DC. 1996. Most professional development 
took place during the school year. Teachers were paid their regular salaries and schools received paid substitutes. 

^ Bouncy Bee Learns Letters and Words. The Children’s Writing and Publishing Center. Combining Sentences Series. Cornerstone Language and Math Series. Expl- 
oring Math Concepts Series. Exploring Math with Manipulatives Series. Exploring Measurement, Time. & Money Series, Math Concepts Series, Math and More. 
Math Practice Series. Parts of Speech Series. Primary Editor Plus, Punctuation Series. Reading for Information Series. Reading for Meaning Series. Skillsbank 96 
Reading & Mathematics, Spelling Series, Stories and More I & II. Student Writing Center. Touch Typing for Beginners. Vocabulary Series, Writing to Read 2000. 
Writing to Write Series. 

^ The program as fielded in 1 990-91 . emphasized basic skills, and was targeted at the elementary grades. As a result, the software, especially the launch versions 
selected from the late 1980s available libraries, emphasized more drill and practice than would be the case in an initiative with access to current or future materials 
or one pointed at different policy goals. 
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Jostens Learning Corporation offers courseware that they believe addresses multiple instructional philoso- 
phies. The courseware allows the teacher to provide individualized Instructional programs based on students' 
needs. The products are designed to enhance critical thinking skills through lessons for the reading, mathe- 
matics and writing curricula. The management system provides an assessment component with a variety of 
reporting options. 

In practice, the implementation of both vendor solutions has become very similar. Both have made a strong 
effort to adapt their materials to the needs of the teachers and students in West Virginia. Both vendors 
provided staff development specifically designed to address issues targeted by the BS/CE program. Both 
vendors provided correlation matrices to the texts on the West Virginia adoption lists and to the standard- 
ized assessment tool selected by the state. 

To fund this initiative. West Virginia used a per-student entitlement strategy rather than a grant strategy. Thus, 
^//elementary schools received hardware, software and training, notjust schools that successfully completed 
grant applications, as in many other jurisdictions. 

The eight-year consistency with which the legislature has passed appropriation bills to support BS/CE 
purchases is equally remarkable. Funding has been about $7 million per year. The state's policy choices 
concentrated resources on a relatively small number of classrooms and that, in turn, enhanced the likelihood 
of a ratio of students to computers that would reach a critical mass. Second, the state invested in its teach- 
ers as seriously as it did in technology. Third, it instituted policies that simplified supervision, training and 
maintenance (the single hardware contract and the two vendor choice for software). Fourth, all schools were 
included and all elementary teachers were assured of their place on the implementation schedule. Finally, 
the legislature provided consistent and sustained funding. The net effect of those choices was a tightly 
focused program. 



1 .IB What Is BS/CE to Students and Teachers? 

The reality of any program differs by the participant's perspective. 

1.1 B1 Students 

For most students, BS/CE represented only two of the three components of the initiative: hardware and soft- 
ware. The conventional wisdom is that change is resisted, but 90% of the state's students thought these new 
technologies in the school were easy to use. Seventy-eight percent of West Virginia's fifth-grade students 
thought that the BS/CE computers had helped make them more successful as students. Two-thirds reported 
that they liked using computers "a lot"; more than half believe that computer technology is a "new basic" 
(two-thirds of the teachers have reached this conclusion). 

The fifth-grade students reported the amount of time they spent working on the computer each week, with 
the majority spending an hour or more a week; 10% of the students spend more than two hours a week. 
Nearly a quarter of the students (22%) work on the computer every day and an additional 36% spend time 
on the computer three or four days of the week. (See Figure 1 .) While BS/CE was intended to boost basic 
skills acquisition, the students inevitably learned about computers at the same time. After involvement in 
BS/CE, 43% report that they know a lot about computers. 
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Student attitudes and comfort with BS/CE and computers in general came from teacher encouragement, 
modeling, and support. Three-fourths of the students said they were encouraged by their teachers to 
use computers. 



Figure 1. BS/CE Computer Use (Student Self-Reports) 
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1.1 B2 Teachers 

Most teachers understood BS/CE as a three part initiative: hardware, software, and technology develop- 
ment, all directed at improving basic skills in students. The initiative sparked collaborative teaching practice, 
which was documented in our one-on-one interviews, case studies, and in the surveys. 

For instance, in West Virginia we saw BS/CE teachers opening classrooms early to accommodate students 
who wanted to use the rooms' computers. We saw other teachers taking pages of student test data off the 
printer and talking with each other about their differing perceptions of the students whose records were being 
printed. In another school, we interviewed two teachers from the same grade level who worked together 
using BS/CE because they believed that each of them was strongest In a different dimension. One created 
a series of math lessons customized to local traditions; the other wrote the beginning of a movie script that 
the students had to complete for a writing project. These activities were used by all of the students in the 
two classes. 




Our procedures allowed us to determine what attitudes were most related to student achievement and that 
discussion extends the picture of BS/CE as it was perceived by the teachers. A few facts pre-figure those 
results: first, almost half the teachers (46%) report that their enthusiasm for incorporating computers into 
their instruction has increased over the years. (See Figure 2.) Ninety-two percent of the teachers concluded 
that instructional technology was not just another fad. 

Figure 2. Teacher Enthusiasm For BS/CE 
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Neither was the enthusiasm of the State's teachers diminished by the perceived requirement that they par- 
ticipate. Though BS/CE was technically not a mandate, when asked why they got involved, nearly half (49%) 
of the teachers said that initially they were "responding to a state-mandated requirement." But when we 
asked why they continued to be so involved, the percent reporting this 'compliance' motivation dropped to 
24%. Teachers came into BS/CE because they were compelled to, and they stayed because they believed 
in what it could do for them and their students. Fifty-eight percent of all teachers surveyed reported, "I feel 
competent guiding student learning through the use of Basic Skills computers." 

In addition to the enrichment and supplementary ways that teachers used BS/CE, they also used BS/CE as 
it was intended, for basic skills. In terms of the most to least frequent use by instructional purpose, teachers 
reported a mean of 4.40 for mathematics use and 4.06 for English/language arts use on a scale of 1 = None 
at all and 5 = Very much. Asked to estimate the amounts of time that students spent learning either mathe- 
matics or English/language arts with BS/CE computers, the majority of teachers chose the maximum time 
estimates possible. (See Table 1.) 



Table 1 



Teacher Estimates aTfStudent Ebmputer Use by Curriculum Areas 




Teacher Estimate of Amount of Student Use 


Curriculum Area 


Percent 
Very Much 


Percent 

Some 


Percent 

Middle 


Percent 
a Little 


Percent 
None at All 


Mathematics 


62 


26 


9 


2 


1 


English/language arts 


57 


24 


16 


3 


1 
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One of the objections to computer-based learning is that it takes time away from higher order, more creative 
tasks, We asked the teachers whether they agreed or disagreed with this point of view: "When students 
learn using computers, they typically reproduce knowledge rather than construct it,” One-fourth agreed, 
but 31% thought that computers were prompting more advanced knowledge construction in their classrooms. 

The same teachers described their own preferred roles as teachers in a more constructivist light ("coach” or 
"colleague” in the construction of knowledge) rather than in the traditional "source” or "master” of information, 

1. 1C Outcomes 

West Virginia's recent history of educational achievement is encouraging. Test performance improved over 
the BS/CE years. For example, after the technology enhanced cohort arrived in that grade, statewide third 
grade CTBS^ scores went up five points. Prior to that time, those scores had risen about 1 ,5 points per year, 
six points in four years. One interpretation of those trends is that the State's improvement trajectory was 
sustained and accelerated by BS/CE students. 

The BS/CE cohort's fourth-grade reading scores (1997) are reported to be the second highest among 
southern states (only one point behind North Carolina), On a national basis, if the achievement scores of 
various states are "corrected” by income, that is, if the unearned increment of school achievement that 
states with high per capita income enjoy from the support that privileged families give their children's learn- 
ing, then West Virginia's test scores improved more than those of any other state. In terms of per capita 
income, West Virginia is in 40th place: in achievement, it is in 17th place. Additionally, student attendance 
improved and early school leaving declined. 

The balance of this report details the relationship between the BS/CE initiative and the achievement out- 
comes which can be associated with that initiative. 



1.2 Research Methods 

We know that these two events — a statewide, comprehensive instructional technology program 
and improved achievement scores — happened at the same time. This analysis tests the extent to which 
West Virginia's achievement gains are related to the BS/CE Instructional technology initiative. 

The importance of research methods goes beyond academic concerns. Policymakers at every level have been 
investing in instructional technology and have a right to know, "Does it work?” And, at every level, different 
functions compete for scarce public dollars. Should money be spent to hire more para-professionals or to 
extend the school day? Should we build new facilities or wire existing ones? Research evidence does not 
" make decisions” : policy Is shaped by the Interests of various publics, by compromises and accommodations 
and by the momentum of existing arrangements. The evidence of research is only one part of decision 
making but, in Its absence, decisions are made solely on grounds external to what best helps children learn. 



^ CTBS, Calilornia Test of Basic Skills, 
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Thus, the Milken Exchange on Education Technology commissioned research that would have several 
policy relevant characteristics. First, because states have the reserved power to determine schooling policy, 
they wanted statewide evidence. Second, because test scores are one central metric of school outcomes 
and children's accomplishments, they required an analysis of educational achievement, using conventional 
state level assessments. Third, because of recurrent skepticism about instructional technology®, a well con- 
structed analysis of results was needed to inform the public discourse and professional practice. 

The hallmarks of a good evaluation design include: a sample selected to represent the population being 
studied; data that are valid, reliable, and from multiple perspectives; students as the unit of analysis; 
multiple forms of implementation documentation; and, the continuing cooperation of the group studied. 

The first step in the analysis was a series of meetings to establish the feasibility of West Virginia as a study 
site. With the cooperation of key state leaders secured and with access to data assured. Interactive, Inc. was 
able to select a sample, gain entry to schools to administer paper and pencil surveys to all fifth-grade 
students and all teachers; interview all fifth grade teachers, all principals, and selected students; and attach 
Stanford 9 achievement data for two years to all students in the sample. 



1.2A Sample 

Since the purpose of this study is to examine the link between technology and student achievement, the unit 
of analysis is the student. Because we need to document specific implementation, teacher attitudes, and 
other variables in relation to each student we studied, a random selection of students statewide was not fea- 
sible within the time and cost restraints of the study. Additionally, because this is a retrospective longitudinal 
study which collects up to five years of data on schools, classrooms, and students, we needed to use our 
resources to insure as much depth and accuracy in these data as possible. Resource considerations pro- 
hibited both local data collection in every school in the state (or even a random selection of a representative 
sample of schools) and a random sample of students, since the latter choice would require the research team 
to collect data in each student's school. Thus, we used schools as an initial stratifier for the sample. 

In order to assure that the schools selected would provide a representative sample of West Virginia BS/CE 
students, we selected 18 schools as the initial stratifier from which we would study all students in those 
schools. The schools were selected with the help of a state education advisory group based upon achieve- 
ment, perceived BS/CE intensivity, geography, vendors, and SES. 

Based on 1995-96 3rd-grade CTBS scores, the 18 schools selected for study range from high to low in 
school level achievement. Furthermore, achievement varies naturally among students and that is appropriate 
since the unit of analysis here is the student. 

Our next concern in sample selection was to find varying student technology experiences. Consensus among 
West Virginia state officials, software vendor consultants with regular access to the schools, and current 
research on student technology experience is that student technology experiences vary as much within group 
(across students in a school) as among groups (across schools). Therefore, selecting schools based on high 
vs. low technology is not necessary to insure a wide variance in experience and attitudes. 

® The skepticism is often a by-product of the competition for scarce resources. In the struggle to get budgets for their preferences, the partisans of any initiative will 
question the evidence on which the competition bases its claims. 



However, as an additional safeguard, we chose schools which, based upon the judgements of both 
West Virginia Education officials and the software consultants who are in those schools regularly, were 
perceived to have a range from most to least technology practice. 

Another criterion in the selection of the schools was geography. State officials believed this an important vari- 
able for sample inclusion — both in terms of variance in achievement scores and in local factors which may 
have an impact on technology use. As a result of discussions with the state education advisory group, we 
divided the state into four geographic areas which represent distinct geocultural and educational variables: 
Northern panhandle (adjacent to Ohio and Pennsylvania), Eastern panhandle (essentially a suburb of 
Washington D.C.), South (rural and adjacent to Kentucky and Virginia), and South Central (the capital. 
Charleston and its surrounding counties). 

An additional and convenient geographic stratifier in selecting the sample is the eight Regional Educational Service 
Agencies (RESA). The eighteen schools for study come from all four geographic areas and all eight RESAs. 

Part of the policy significance of this analysis is West Virginia's unique statewide implementation strategy 
which required a choice between two software vendors: IBM and Jostens Learning Center. Additionally, the 
majority of the implementation assistance came from the software vendor consultants, not the state. 
Therefore, software is an important consideration in reconstructing implementation and individual experi- 
ences. We selected schools so that the proportion of Jostens and IBM programs used by students would 
be proportional to their use statewide. As a result, twelve of the schools selected were Jostens' schools and 
six were IBM schools. 

Based on West Virginia Department of Education data on the percentage of students receiving free and 
reduced lunch, the 18 schools also range from high to low in terms of socioeconomic status. 

The result is a sample of 18 schools which vary with respect to: 

achievement test scores, 

BS/CE and technology experience, 
geography, 

software vendors, and 
community SES. 

The next step in the sample selection was to determine which students from the 18 schools would be stud- 
ied. Prior to the 1996-97 school year in West Virginia, the CTBS was administered for the 3rd-, 6th-, 9th- and 
llth-grade students only. Beginning in the 1996-97 school year, the Stanford Achievement Test (9th Edition) 
was administered to all students in grades 3 through 11. Therefore, the current 5th-grade students are the 
only students in the state of West Virginia for which the state has three consecutive years of achievement 
test data and who have had any BS/CE exposure. Because of BS/CE's follow-through phasing, the current 
5th graders are the students for whom technology has been most available. Therefore, we documented the 
experiences of all current 5th-grade students from their first year of experience with BS/CE through their 
current experience. 

Our student sample included all 950 students in 5th grade in the 18 stratified schools. To generalize to all 
K-6 students in West Virginia from K-6 (N=161,231) at a 95% level of confidence, we would have needed 
data on 384 students. This sample, then, is representative both in that it includes students with a range of 
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achievement, technology experiences, geography, and SES, and it is nearly three times more students than 
necessary to assure confidence in the accuracy of the analysis. 

1 .2B Data Collection: Inputs 

Because this study is an examination of the relationship between technology use and student achievement, 
we had to determine how we would document technology use as well as other variables which might influ- 
ence use. The technology factors included: BS/CE use; other technology use; attitudes toward technology; 
home technology practices; and other demographic and school variables. 

To determine student use and attitudes, we used student and teacher survey and interview data. Input data 
were collected in the following ways. 

Surveys of 950 5th-grade students, using a 33-item paper and pencil survey. This survey focused 
on BS/CE use; attitudes toward schools, computers, and technology in general; and other 
factors which relate to technology use and learning. This survey allowed students to record their 
technology experiences from kindergarten through their current school year. 

^ Surveys of 290 third- through fifth-grade teachers, using a 99 Item paper and pencil survey. 
We administered this survey to all the teachers in all 18 schools in an attempt to capture data from 
those teachers who had taught our fifth-grade student sample for the past three years. 

Interviews with all fifth-grade teachers in the 18 sample schools. 

^ Interviews with all principals in the 18 sample schools. 

Interviews with selected early-grade teachers in the sample schools. 

^ Analysis of documents in each school related to technology planning and implementation. 

In addition to the data from students, we asked teachers about student use as well as their own attitudes 
and practices. Ninety-one percent of the teachers were female. Sixty-three percent have been teaching for 
more than 21 years; 29% for 1 3-20 years; 5% for 6-1 2 years; and only 4% have been teaching for 5 or fewer 
years. 

We Interviewed all the fifth-grade teachers and selected early-grade teachers in each school to try and under- 
stand more completely how BS/CE was implemented in their classrooms and what this meant for how 
students learn. These teachers provided current use data for each student on BS/CE and their own past 
curriculum use of BS/CE. 

During the winter of 1998, each school was documented, on site, by an Interactive, Inc. field researcher. To 
determine if other initiatives were happening or If political or internal issues might have affected both imple- 
mentation of BS/CE and student involvement, we Interviewed all principals, fifth-grade teachers, and selected 
early-grade teachers in the 18 schools. In addition, we analyzed classroom, school, and state documents to 
complete the case descriptions. 

1 .2C Data Collection: Outputs 

Once we collected the student use data and the BS/CE descriptions, we turned to measures of achieve- 
ment. Prior to 1996, the West Virginia Department of Education administered the CTBS to all students in 
grades three, six, nine and eleven. Beginning In the 1996-97 school year, a new statewide assessment pro- 
gram was implemented with all students in grades 3-1 1 taking the grade-appropriate Stanford-9 achievement 
test. Thus, we had two years of Stanford-9 test score data to consider, and an opportunity to consider the 
relationship between gain scores and BS/CE. 



19 



( 23 / 

\ 



The scores that were used in the analysis were scaled scores on the Stanford‘9 achievement test. Because 
scaled scores are normed against a nationally representative group, they are appropriate for comparison pur- 
poses and for the computation of gain scores. The Stanford-9 achievement test is divided into a number of 
subtests, and where appropriate, those scores are reported. Our focus is on gains in "basic skills," a score 
computed and reported by the West Virginia Department of Education for comparison purposes, and a 
score that measures the math, reading and language arts areas that were the focus of BS/CE. Scores 
on each of the three areas could range from 400 to 800. For 1998, the range of "basic skills" scores in the 
state was from 547 to 766 and the average gain score was 14 points. 

In this study, we computed gains in Stanford-9 results to measure student achievement from one year to the 
next. Gain scores represented the difference between the scores on a basic skills combined measure of math 
and reading and language arts from the 1996-97 test to the 1997-98 test. Thus, achievement was repre- 
sented by Stanford-9 scores on reading, language arts and mathematics for all students in the sample for 
1997-98 and 1996-97 

1 .2D The Model: Access/ Attitude/Training 

To determine the relationship between the BS/CE inputs and the achievement outputs, we analyzed our data 
using a model that includes access to software and computers, attitudes toward technology, and teacher 
training and involvement: 



The Model: Access/Attitude/Training 

(1) Software and Computer Availability and Use (software focused on basic skills, computer 
availability, availability of other software and technology programs, time using basic skills software) 

+ (2) Attitudes Toward Computers (student attitudes toward computers, teacher attitudes toward computers) 

+ (3) Teacher Training and Involvement in Technology Implementation Decisions (teacher profes- 

sional development and involvement in implementation decisions) 

= Predicted Change in Acheivement Test Scores 

We speculated that Stanford 9 achievement gains would be the greatest in schools with the highest amounts 
of the model components. 

1.2D1 Software and Computer Availability and Use 

This component includes four items: basic skills software, hardware availability, availability of other 
software and technology, and time spent using basic skills software. 

(1) Basic Skills Software. The development of interactive software to teach basic skills at each grade level 
and the availability of that software to students and teachers Is likely to result In student use of the software. 
Data for this variable comes from observation and the West Virginia Department of Education. 

(2) Computer Availability. The more computers that are available to students and the more students report 
that there are computers available when they want them, the more likely it is that student computer use and 
the technology implementation can occur. Data for this variable comes from a yes/no response on the 
student survey. 
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(3) Availability of Other Software and Technology. This is a school wide variable which was measured in the 
site visits by inquiry from principals, teachers, and students as well as by observation of technology initia- 
tives and the collection of documents submitted to the State Department of Education. 

(4) Time Using Software. This component of the model includes both time spent on BS/CE and time spent 
using other software. We asked teachers to describe their students' use of computers and BS/CE. Their 
answers resulted in a factor which combined the responses to six questions in the teacher survey and 
represents the amount of time that teachers estimate students use the technology in math, reading, and writ- 
ing. Total scores on this factor ranged from 5 to 15, moving from none in math, reading, and writing to 
60 minutes or more in math, reading, and writing. 

A series of questions to students resulted in a similar student-reported technology use factor. Teacher and 
student measures of time were closely correlated. Total scores on this 18 point factor ranged from 4 to 18, 
from zero in math, reading, and writing to 60 minutes or more in math, reading, and writing. (See Figure 3.) 



Figure 3. Time Using BS/CE Computers 





5th Grade Use Factor 5th Grade Use Factor 

(Based on Student Estimates) (Based on Teacher Estimates) 

The second "use" component focused on technology use other than BS/CE. We predicted that time using 
any instructional technology leads to increased skill and comfort with computers and the technology and 
basic skill learning of the primary initiative. The use of technology independent of BS/CE was documented in 
the qualitative case studies in each school and each classroom. The case study data were then recoded into 
a four option variable ranging from "no additional technology" to "extensive additional technology." 
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1.2D2 Attitudes Toward Computers 

The attitude component includes both student and teacher measures. 

The student attitude factor combined the results of student responses to two survey questions: the scores 
ranged from two to four. The model predicts that the more positive students feel about computers and tech- 
nology, the more likely they are to use BS/CE software. 

Two factors represented teacher attitudes. The first was a factor combining the results of questions from the 
teacher survey focusing on teachers' comfort level and confidence in using computers. The second is a 
factor that combined the responses to two questions which tapped teacher willingness to continue learning 
about computers and technology. 

1.2D3 Teacher Professional Development and Involvement in Implementation Decisions 

This component includes both the training that teachers received and their involvement in the planning and 
implementation of BS/CE. 

Teacher training is the amount of time teachers spent learning on the computer on their own. We predicted 
that the more teachers are involved in development of the technology implementation plan, the more likely 
the implementation is to be carried out. We measured this involvement using questions on the teacher 
survey. Teachers responded to a five point Likert scale. The results were clarified through teacher interviews. 

1.2E Analysis 

After factor analysis had established the reliability and validity of the three components just described, the 
relationship of BS/CE to student achievement gain scores on the Stanford-9 was analyzed through multiple 
regression analysis. 

Interactive, Inc. circulated all reports in draft to officials from the West Virginia Department of Education for 
correction of matters of fact and for comments on our interpretations and conclusions. [Nota Bene: 
responsibility for this analysis, its conclusions and recommendations rests solely with the authors 
and does not reflect the policies of either the West Virginia Department of Education or the Milken Exchange 
on Education Technology.] 



1.3 Summary 

The net result of the state's initiative, its accomplishments and the procedures for this analysis makes 
West Virginia a potentially illuminating case study of innovation in instructional technology. The implementa- 
tion was statewide yet tightly focused. The outcomes were significant in terms of policy and of statistics. And 
the events and the outcomes have been comprehensively assessed. 
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2-0 RELATIONSHIP OF B5/CE TO 5TUOENT ACHIEVEMENT 



To examine the relationship between the BS/CE experience and student achievement we computed gain 
scores on the Stanford-9 for each student from 1996-97 to 1997-98, Additionally, for each student we 
gathered data for each of the regression model components which measure: 

Software and Computer Availability and Use, 

Attitudes Toward Computers, and 

Teacher Professional Development and Involvement in Technology Basic Skills Implementation Decisions. 



2.1 BS/CE's Effect on Student Achievement 

With the student as the unit of analysis, we examined the relationship between how much of each of the 
model variables that student had experienced and her or his gain scores on the Stanford-9. 

The more of each of the model components that the student experienced, the higher the gain score on the 
Stanford-9, Specifically, the BS/CE technology regression model accounts for 11% of the total variance In 
the basic skills^ on the Stanford-9 achievement test. (See Table 2.) 



Table 2 



lyiodel Suiiim gain scores of the lift students. 


r 


r^ 


adjusted r^ 


std error of 
the estimate 


.331 


.11 


.094 


14.8317 



Thus, student gain scores can be partially explained by a model composed of factors that describe the 
overall BS/CE experience, an impact which is powerful and practically significant; it Is also statistically 
significant at more than the .001 level. More to a practical point, an 11% improvement in test scores would 
be welcomed by most students, parents and educators. 



2.2 Policy Significance of the BS/CE Effect on Student Achievement 

Analysis indicates that test scores of our student sample improved from 1996-97 to 1997-98. Conventional 
procedures ask, "What percentage of that whole gain can be explained by the BS/CE initiative?" Our data 
indicate that 1 1 % of the reason why student scores increased was because of BS/CE. 

What does this mean? Is 11% more than just statistically significant? Does It have any practical value to 
students, educators and policymakers? We think so. To understand what the 11% of explained variance 
means, it is important to understand all the influences on student achievement, the most significant being 
family background. 



^ An average of the students’ Total Math, Total Reading and Language scores 
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Family background explains a great deal of why some children do better than others. Since James Coleman's 
1965 research, we have known that children's achievement in school is conditioned more by what they bring 
to school than by what schools are able to do with them. Consider: 

(D)espite the wide range of diversity of school facilities, curriculum and teachers, and despite the wide 
diversity among student bodies in different schools, over 70% of the variation in achievement for each 
group is variation within the same student body (James S. Coleman, et al. Equality of Educational 
Opportunity. Washington DC., US Department of Health, Education and Welfare, 1966, p 77). 

In thinking about schools, this difference between what schools can and cannot influence is intuitively and 
practically obvious. Imagine trying to search out the explanations for gains in test scores between two groups 
of children. One group lives in houses where the following are customary: daily reading that is modeled by 
adults and encouraged for children; availability of books and magazines; frequent use of computers and com- 
puter games; after school learning and enrichment activities; parental or other adult supervision of homework; 
and, visits to museums, art galleries, and science sites. 

The other group of children live in homes where: there are no books; reading is not modeled, encouraged 
or frequent; unsupervised television is constant; there is no computer; there is no after school enrichment; 
there is no supervision with homework; and, there are no visits to cultural events of the sort reflected on 
achievement tests. 

To document the amount of learning for both groups of children, an achievement test is administered which 
measures: knowledge of mainstream culture; familiarity with standard English; art and science experiences 
outside the classroom; and the benefit of extra drill and practice In the basic skills. 

It is not surprising that the children with all of the educational and cultural extras provided by the family and 
tested by the assessment do better than the children who do not have the related experiences. This is the 
power of the family and of economic privilege. According to Coleman, the resources of families, the culture, 
the media and the peer group account for 70% of the differences In student achievement. 

If the performance of children depends heavily on the characteristics of the families they come from and if 
we further understand that those family characteristics are for the most part beyond the reach of public 
policy (for example, proscribing divorce, prescribing post-graduate education for parents, and providing all 
families with equal social capital), then we ought to concentrate our attention on the things that are, In fact, 
accessible to influence by school policy. 

If, as Coleman and others assert, 70% of the variation in test score performance relates to family and other back- 
ground factors, that leaves 30% that schools can influence. We call that 30%, "school accessible performance"®. 

This analysis documents that as much as 11% of the gain score variance for one year can be explained 
by BS/CE. Thus, BS/CE explains more than a third of the school reasons why students' achievement 
scores improved. Said another way, of all the factors that can affect a child's learning, about 30% is 
within the school's sphere of influence. The BS/CE technology initiative explained nearly 11 of those 30 
percentage points. 




® The school improvement literature concentrates on the institutional side of this equation, not on the larger family and community surround. Ideas of “school effects" 
and of “within school factors' can be found, for example in Michael Rutter. Barbara Maughan, Janet Ouston, Alan Smith, 15,000 Hours: Secondary Schools 
and Their Effects on Children, Cambridge. Harvard University Press, 1979. See also. Peter Mortimore. Pamela Sammons. Louise Stoll. David Lewis, Russel 
Ecob. School Matters: The Junior Years, Somerset, Open Books. 1988. 
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This effect is reflected in the change in West Virginia s rankings among other states. During the period 
studied, West Virginia moved up the list of states in the nation rank-ordered by school achievement from 
thirty-third to seventeenth best. Since we can establish the fraction of achievement score gain that is related 
to BS/CE, it is reasonable to assign some portion of the credit to that initiative. It is also reasonable to 
believe that the effects of BS/CE, although impressive, are underestimated. 

There are two additional reasons to believe that the reported gain scores underestimate the total effect of 
BS/CE. The 11% of the gain score variation explained by BS/CE was between 1997 and 1998 scores. 
Those gains happened between the fourth and fifth grades and it is very likely that there were earlier increas- 
es, attributable to this initiative, between the third and fourth grades and the second and third grades and so 
on back to the beginning. Thus, the cumulative effect of BS/CE most likely accounts for more than the 11% 
of the single-year gains that we found. 

We also suspect that the gain scores reported are unnecessarily conservative because current hardware and 
software is more powerful than what was implemented in BS/CE s early years. Current technology is likely 
to yield even larger gains. 

If BS/CE can be credited with some considerable effect on statewide test score gains, then did BS/CE 
"make children perform better?" The question is reasonable but on the evidence we have, strictly interpret- 
ed within the canons of social science, we cannot say with certainty. To be able to do so, the State would 
have had to have randomly assigned some students to BS/CE and randomly assigned other students to 
conditions where BS/CE was not available. While that might be good social science, such practices are poor 
and probably unethical public and educational policy. 

Thus, as a consequence of the statewide implementation, it was not possible for us to compare control group 
schools with the BS/CE schools. And, the achievement score record prior to 1990 is insufficiently detailed 
to support a pre/post inquiry. As a result, our s is an analysis of correlations (things that vary together), not an 
analysis of causation®. No one withheld instructional technology from half the children; instead the State 
made BS/CE available to all children. However, as is usual in any initiative, some students experienced more 
of each of the BS/CE technology components than others, so what we have is a situation where all students 
received something and some students received more of some things than did other students. Those 
combinations are not uniform and vary by student, even within the same school or the same classroom. And 
those variations make possible the procedures of this inquiry. 

As important as the procedures of social science research are, the needs of policymakers are also legitimate, 
so it is possible and sometimes helpful to interpret correlational data to suggest causality^®. We have done 
so in this instance, suggesting that the unstandardized betas that we report are indices of the effect of the 
related BS/CE component on Stanford-9 gain scores. (See Section 2.3.) 

We can help with the "What works" question by first, assessing the statistical significance of the numerical 
findings. Big findings may suggest big relationships. Second, we can ask about other explanations for 
the findings. Sometimes those alternate explanations can be disposed of; sometimes they can be understood. 

® The absence of findings Ihaf rise to the level of causation is (he norm in social policy. Virtually no decision in public schooling Is informed by causal analysis 
—not finance decisions, not racial integration decisions, not curriculum or testing or personnel selection decisions. 

Pedhazur and Schmelkin (1996) believe that the unstandardized betas in nonexperimental research can be interpreted as indices of effect if the model is specified 
based upon theoretical foundations and previous research. Because our model is so specified, we maintain that our unstandardized betas indicate effect. 
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In this case, we believe that the several dimensions of BS/CE explain a substantial fraction of the school- 
accessible performance. An explanation of that magnitude is intrinsically worthy. The part that is not explained 
by BS/CE may have to do with initiatives such as statewide standards and testing, or West Virginia's " Unified 
School Improvement Plan(ning)" or school-site accreditation visits coupled to the possibility of probation. 
Empirically determining the effect of those procedures was not within the scope of this work, but we did 
examine nearly 50 other variables for alternative explanations and found none^T (See also "4.0 Additional 
Explanations For Test Score Gains.") 

We believe that reasonable people, experienced with school policy, will find that this analysis supports the conclu- 
sion that instructional technology is a powerful assist to children's achievement^^ and thus to school improvement. 



2.3 Effects of the Components of BS/CE on Achievement 



Public policy has a continuing interest in answers to the "What Works?" question. Here, the whole BS/CE 
program makes a difference. No single component is sufficient to account for the BS/CE-associated 
outcomes and neither, probably, would it have been sufficient to maintain this program for only a few years. 

It is not surprising.that no single component dominates. Putting hardware in a room without training teachers 
or otherwise supporting the integration of technology into the classroom cannot be expected to make a 
difference. It is the cumulative effect of the several variables that compose the model that is important. 

The several factors of our model resemble the several components of BS/CE policy. Statistically, as practi- 
cally, it takes multiple dimensions to make a difference. The model demonstrates that it is software specific 
to the purposes of basic skills achievement, availability of computers, teacher training and involvement in 
implementation decisions, positive student and teacher attitudes toward computers, and time spent using the 
software that lead to achievement gains. 

Taken together, these factors account for 1 1% of the variance in basic skills gain scores at more than a .001 
confidence level (n = 502). 

The three components of this empirically-derived model — access, attitude, and training — are similar to what 
leaders in instructional technology advocate. Unless there is sufficient equipment and opportunity to learn 
(access) there are unlikely to be effects from Instructional technology. And, unless teachers have a chance 
to learn about the technology and how It may make them and their students more successful (training), they 
are less likely to believe it can help (attitude) and, in fact, to use it. 



" For instance, we examined the effects of several student, teacher, and school variables on gainscores in our model creation and testing and found no additional 
explanatory power, for the analytic model, from these variables. Student variables tested include race, sex, age, geography, homework support, familiarity with 
Internet and E-mail, attitudes about school, attitudes about learning, time spent on homework, grades, and other achievement measures: teacher variables include 
attitudes, pedagogical philosophy, experience with computers and software, attitudes toward state initiatives, attitudes about teaching, attitudes about student 
learning, cost to teacher (in time, etc.) of implementation, professional development, homework expectations, and attitudes about families: district variables include 
local political actions, changes in administration, additional initiatives, deployment methods, availability of labs, and colleagueship. 

54% of the teachers thought that BS/CE was instrumental in test score gains. 
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The teachers' confidence in using computers and the amount of time they spend using them are a bundle of 
attributes that may be influenced by experience, professional development and the teachers' perception of 
their principal's leadership. The teachers' attitudes and behaviors also interact with what the students' 
experience — increased use and positive attitudes toward computers. 

We constructed the model first to test the possible relationships between BS/CE and student achievement 
and second to illuminate how BS/CE is probably making a difference with students. Empirically, we know that 
the complex of factors listed above will explain the amounts of achievement gain we have reported. 

Practically, it is likely that BS/CE is having its effects through its ability to impact students through the model 
factors. With students, for example, the availability of computers and teacher encouragement are likely to 
increase computer use and, as use increases so does the amount learned and thus (probably) achievement. 
Similar logic applies to teachers. As BS/CE has rippled through the faculty culture over the years, it has 
changed attitudes and that becomes part of the intervention, part of the effect and part of the success. 

For example, 69% of the students thought computers were as important as "reading and writing." Sixty-six 
percent like computers a lot, only 5% do not like computers. Sixty-one percent report that their teachers 
encouraged them to use computers. 

School reform is often criticized for relying on "The Magic Feather Principle;" the idea that there will be a 
singular solution to complex problems. West Virginia's BS/CE model used multiple interventions to support 
and change multiple functions of teaching and learning. 



2.4 Access/ Attitude/Training Model Effect Size 



As noted in section 1.2D, there are three major components of the technology initiative under study: 

Software and Computer Availability and Use (software focused on basic skills, time using BS/CS, 
use of other computer technology initiatives, availability of computers). 

Attitudes Toward Computers (student attitudes toward computers, teacher attitudes toward computers). 
Teacher Training and Involvement in BS/CE Implementation Decisions. (See Table 3.) 

Understanding the strength of each of the model components is best done by examining the betas. While 
betas are a kind of effect size, they are not a straightforward measure, especially in the social sciences where 
variables are often inter-correlated. However, where a model is well specified, even if the study is not exper- 
imental, it is possible, particularly for policy purposes, to interpret unstandardized betas as indices of effects. 

In our case, all of the components of this model stand as independent variables because there is very little 
correlation among and between them. The correlation between variables ranges from a low of .079 to a high 
of .224 and, thus the model has relatively low multicollinearity among its components. The tolerance for each 
variable in this model runs from .564 to .971, where the value Is the proportion of variance unique to 
each variable. Six of the nine variables report tolerances in the 80th and 90th percentiles, while three hover 
at about the 60th percentile. 
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Table 3 



Access/Attitude/Training Model EfFect Sizes 

Unstandardized Coefficients Standardized Coefficients 



10 

5 



25 



'6 



5 



The meaning of the unstandardized betas teils us how much of an effect the nine variabies that make up the 
three components of our model have on achievement, which in our case means that if all the components of 
the model are in place, achievement increases by between eight and ten months. 



; Hardware and Software iAccess and Use 



B, 



Std. lError 



Beta 



*Sig 



Computer and 
BS/CE Use 



1.453 



.272 



.240 



5.338 



.00 



Software Access 



5.958 



1.470 



.184 



4.054 



.01 



Computer Access 



2.126 



1.347 



.068 



1.579 



.1 



Other Technology 
Access 



.715 



.659 



.051 



1.086 



.278 



Attitudes 



Student Attitudes 
Towards Computtirs 



2.742 



1.222 



.099 



2.24- 



.0 



Teacher Attitudes 
Towards Compute!' 



1.069 



.434 



.123 



2.461 



.014 



Teacher Confidence 



.310 



.584 



.030 



.530 



.5! 



teacher Training: and tnvolvment 



Involvement 



-1.959* 



1.059 



-.097 



-1.851 



.06 



Training 



1.109 



.656 



.095 



1.619 



.091 



’Negative numbers indicated higher teacher involvement. 



In schools that are hard-pressed to find time for all the mandates and all the required classes, time counts. 
In this analysis, "time" has two facets — the frequency with which BS/CE was used in any given year and the 
repeated, cumulated experience of BS/CE over the years. 

The variable that measures reported instructional time in a year accounts for more of the difference than any 
other. The relation between time-on-task and learning gains is commonly observed. Extending the school day 
or year, cutting recess, dropping curriculum topics all make possible increases in time-on-task. What is 
different here is the use of technology to Increase time-on-task. In a direct instruction, teacher-centered 
classroom there are a fixed number of minutes for a teacher to deliver instruction. When students use 
computers to work independently, the amount of student time-on-task can be multiplied independent of the 
limits on the teacher's agenda and availability. 

Teachers make decisions about classroom time but so do state policymakers. BS/CE has been sustained 
over an uncommonly long Interval, seven years. Students reported increased computer use every year from 
kindergarten through the fourth grade. For example, the percentage of students who described themselves 
as using "BS/CE computers a lot" increased from 20% in kindergarten to 53% in the fourth grade. Similarly, 
the earlier in their school careers that students began using computers, the more likely they were to 
continue Intensive use. (See Figure 4.) 



BEST COPY AVAILABLE 
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Figure 1 Use 




Kindergarten 1 st Grade 2nd Grade 3rd Grade 4th Grade 



And, almost half the teachers became more enthusiastic about BS/CE as time passed. Neither would have 
happened without the willingness of the state's policymakers to keep instructional technology at or near the 
top of the priority list. We believe that time within the school day and year AND time across the student's life 
in school are part of the context for the achievement score gains. 
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2.5 The Distribution of Achievement Gains Across Groups of Children 

The benefits of BS/CE fall differently by type of child. Gender and SES effect how well the model predicts 
achievement. 

2.5A Growth for All Students 

The amount of the gain score changes explained by BS/CE are significant in both policy and statistical terms. 
But, gain score differences focus the amount of change in one child or group compared to the amount of 
change in another child or group. What they do not reveal is whether or not the whole group was getting bet- 
ter and, In West Virginia, It was. In addition, students were able to recognize the relationship between their 
computer experience on BS/CE and their learning gains: Seventy-eight percent of the fifth graders thought 
that computers had helped make them better students. 

The effect of the BS/CE technology initiative on student achievement can be Illustrated by thinking about 
a bank of elevators. All the elevators are carrying passengers up, but their speeds may vary. In addition, pas- 
sengers get on at different floors and get off at different floors. All passengers get someplace in the elevator; 
they start at different places and stop at different places, and they all go up. It is important to see that, although 
they start in different places and some get to the top faster than others, all the passengers are being lifted. 

2*5B Equity Effects 

The data indicate that BS/CE helped all children learn and it helped the neediest children the most! (Special 
education students reported the same amount of time on BS/CE computers as regular education students). 
Those children without computers at home made the biggest gains and that is good news for public policy — 
the children who most need the public school can be helped by this sort of policy. The schools we studied 
were chosen to represent a range of socio-economic characteristics. Some were urban, suburban and rural. 
Some schools served decidedly higher income families than did others. 

The "Digital Divide” is mapped by those West Virginia students who do (62%) and do not (38%) have 
computers at home. Those without computers at home gained more in: 

^ total basic skills, 
total language, 
language expression, 
total reading, 

reading comprehension, and 
^ vocabulary. 

The Access/Attitude/Training Model explains more of the basic skills gain scores for students who report 
lower grades than for students who report higher grades. For students who report receiving grades of C, the 
model explains 19.3%, versus 15.6% for students who report grades of B, and 10.7% for those reporting As. 

Thus, BS/CE is more strongly related to gains for students who have less family and social capital and for 
students who do less well in school. 
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The chronic challenges of race and education condition us to expect lower achievement for some children 
than others. It is progress that in West Virginia, there were no differences in gain scores between white 
students and non-white students. On 1998 achievement, there were no differences overall, but white 
students achieved higher scores on listening, vocabulary and reading. 

2.5C Gender 

One regularity of both the technology and the gender literatures is the relative disadvantage of girls with 
respect to technology. It is progress that girls and boys had the same access and the same use of comput- 
ers in West Virginia. 

While there are no differences in the amount of use between girls and boys, the girls were more likely to see 
computers as a tool and the boys as a toy; boys are more likely to report that computers are fun. 

Equal access and use of computers and software by girls and boys in West Virginia proved important in terms 
of girls' comfort with and attitudes toward computers. The more years that girls report having used com- 
puters, the more they like them and the more they report knowing about them. Unlike many girls, the girls in 
West Virginia are more likely than the boys to say they know more about computers than do boys. In addi- 
tion, the girls reported finding computers more accessible than their teachers to their particular learning 
needs; girls are more likely to consider it easier to learn from computers than from their teacher. This finding 
might indicate that computers, unlike some teachers, respond in the same ways to both girls and boys and 
that either sex can ask questions, linger, or repeat activities on a computer. 

In terms of gain scores, there were differences in only two areas related to gender — girls gained more in 
social studies and boys gained more in spelling. In math and reading, there were no gender differences. 
However, for the actual 1997-98 scores, girls did better in language, reading and study skills and boys did 
better in spelling. There were no gender differences in mathematics in 1998. 

The Access/Attitude/Training Model is a more powerful predictor for boys than for girls. The model explains 
16.1% of the basic skills gain score for boys, compared to 11% for girls. This is probably because of the dif- 
ference in 1998 achievement on the language subtests on the Stanford 9. Because girls did better than boys 
on those language subtests, and because language and reading were improved by using the BS/CE tech- 
nology, the more boys used BS/CE, the more likely they were to improve on Stanford 9 measure of language. 



Fifty-one percent of West Virginia's teachers are confident in using computers in their teaching, only 19% are 
not. Two-thirds say they are very interested in computers: a third believe that technology has ‘'empowered" 
them as teachers. Half the teachers thought that technology had helped " a lot with West Virginia instructional 
goals and objectives." 

We asked teachers to grade themselves according to how skillful they were in using computers for various 
functions. Typically, the closer the activity is to the "performance art" core of classroom work, the lower the 
marks teachers give themselves. While West Virginia teachers select "word processing" as their most 
skillful application, the next highest rated functions are central to teaching. The table below includes com- 
parison values from another large-scale analysis of teacher use of technology. 



2.6 Teachers and BS/CE 
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The skill levels in delivering instruction, planning lessons, managing paperwork, and word processing of 
West Virginia's teachers who had computers in their classrooms were significantly higher than those who had 
computers in the labs. (See Table 4.) 



Table 4 



Teacher Self-Estimates df Skill Levels for Various Functions 

. -(N = EBm 


Function 


Mean Score* 
West Virginia 


Mean Score 
New York 


Delivering Instruction 


3.00 


2.48 


Planning Lessons 


2.52 


2.48 


Networking 


2.42 


2.04 



*Range: 1 = low, 5 « high. The New York data are from; Dale Mann and Edward A. Shafer, "Technology and 
Achievement" The American School Board Journal, v 184, n 7, July 1997, pp 22’23 based upon a 1996 study done 
for the Mohawk Regional Information Center, Verona, New York by Charol Shakeshaft, Dale Mann, Robert Kottkamp. 
and Mike Mussi, Interactive, Inc., Huntington, New York. 



Simple correlations suggest that collaborative learning and constructivist ideas were most likely to occur in 
the classrooms of those teachers who report the highest computer confidence level and the most time using 
computers In their instruction (both factors that contribute to the BS/CE Access/Attitude/Training Model). 

We asked some questions to test possible negative attitudes about computers. (See Table 5.) 

Table 5 



Teacher Reactions to Purported Negative' Efifects from 
Eomputer-Related Technology* 

■ i ' : . : 


Purported Effect 


% Disagree 


% Agree 


"Computers... 






...take time from direct instruction" 


61 


19 


...make class management more difficult" 


65 


13 


...promote frustration and aggravation" 


51 


20 


...require more planning" 


39 


28 


...take too much time to use" 


63 


11 



*Mid-point, "3," responses not reported. 



The most frequently chosen comment — that enhancing instruction by using computers 
requires more planning time — reflects the reality that adding new techniques takes time. 
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3.0 IMPLEIVIENTING B5/CE 



3.1 Critical Mass 

Change in schools is generally in small increments (and little interventions are easier to ignore). BS/CE was 
an exception, at least in part because the grade-by-grade follow-through strategy concentrated significant 
resources on tightly defined targets. Prior to the 1990-91 inception of BS/CE, the average school owned a 
total of 14 computers. If half were then used for administrative/clerical functions and two were in the library, 
that meant that five classrooms might have one (Apple) each. Contrast that with the arrival of four machines, 
a printer, networking capability, software and professional development multiplied by the number of class- 
rooms at that year s target level — all showcased at the beginning of the school year. BS/CE was a big deal, 
especially for the rural schools. 



3.2 Fidelity of Implementation 

BS/CE was also fielded as it was designed. For school improvement in general, the most common explana- 
tion for a lack of outcomes is a lack of implementation — the training is planned but canceled, the books are 
purchased but not required, people are hired for one purpose and end up doing something else. That does 
not describe BS/CE. In the West Virginia case, the districts and the schools did use the discretion that they 
were accorded in software selection and in style of deployment (labs/centers versus distributed versus com- 
bined). But the major points are that not only does BS/CE have effects associated with its implementation, 
it was implemented. 

Our field work included extended visits to 18 schools. The BS/CE computers were where they were sup- 
posed to be (in classrooms or in labs/centers). When teachers were asked to describe how they used the 
BS/CE machines, 80% responded, "reinforcement of the standard curriculum," that is, applications tightly 
targeted on Basic Skills. Although there were some complaints about machines arriving before training or the 
unavailability of technical support, the amount of 'noise' in the West Virginia system was tiny compared to 
the scope and fidelity of the effort. BS/CE was fielded as it was designed. 

BS/CE's implementation was more influenced by the State than is the usual practice. True, the initial design 
was done by a group that included teacher representation but It is also the case that teachers were required 
to be trained (and supported with salaries or stipends for that). True, counties could choose their software 
suppliers but only between IBM and Jostens and all hardware was IBM. In the final analysis, the reality of 
BS/CE came down to the choices of teachers behind closed classroom doors, but those classroom deci- 
sions were systematically channeled, encouraged and supported In very particular directions. 

3.3 Labs/Centers versus Classroom Distribution 

Schools could Choose how to deploy the BS/CE computers — concentrated in labs (6 of the sampled 
schools, 293 students), concentrated in classrooms (5 schools, 273 students) or a combination of lab- 
plus-classroom distributed (7 schools, 380 students). The deployment choice Is significant for policy and 
crucial to teachers. (See Figure 5.) 
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Figure 5. BS/CE Computer Deployment 





Combination 



Distributed 



Lab-Based 



Students who had access to BS/CE computers in their classrooms (the "distributed" pattern) did signifi- 
cantly better than students who were taught with BS/CE equipment in lab settings. The students taught in 
the classroom pattern had higher gains in overall scores and in math. They also scored higher on the 1998 
tests than did those in labs. (There were no differences on scores by source of software.) In distributed class- 
rooms, the BS/CE efforts account for 19% of the variance In test scores. 

Teachers who had computers in the classroom reported higher skill levels in managing instruction, planning 
lessons, delivering instruction, and word processing. 

The power of classroom integration belies the low student:computer ratios reported by teachers in schools that 
used labs as centers. There, 77% of the teachers reported 1 :1 student:computer ratios but apparently omitted 
to notice that this only applies during those minutes per week that their students are assigned to the lab. 

Teachers who had BS/CE computers in their classrooms (the ’distributed' pattern) reported more time using 
BS/CE computers for reading, math and writing instruction than either of the other two patterns. Sixty-one 
percent of the teachers with access to computers in their own classrooms said they were confident in using 
computers in their teaching compared to only 43% of the teachers who took their children to the lab for 
instruction in, about, or with computers. 



4.0 ADDITIONAL EXPLANATIONS FOR TEST SCORE GAINS 

West Virginia's experience with Instructional technology Is important because of the gains that can be asso- 
ciated with BS/CE. But the State was also changing other aspects of schooling at the same time. In 
understanding the significance of instructional technology, it is important to inquire into those other initiatives. 

BEST COPY AVAILABLE 
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During this period, the State spent $430 million to renovate 470 school buildings. Sixty-eight new schools 
were built. (The state also closed 330 schools in six years, down to 840 buildings.) It is not likely that the 
newness of the physical plant directly impacts learning (Armor's research indicates a slight negative relation). 
But school attendance went up and drop-outs went down in the 1991-1997 period. It may be that a more 
attractive place attracts more attendance and thus increases exposure to teaching and the possibility of 
Increased learning. 

In 1989, the year the State legislature passed the BS/CE enabling legislation. West Virginia teachers were 
the 49th worst paid in America. In 1990, there was a statewide strike of teachers. Beginning that year, all 
teachers received a $5,000 pay Increase''^ spread over three years: West Virginia's statewide average 
teacher salaries have risen 15 places among the states. 

When asked to rank order three phenomena In terms of their probable effect on student learning, 48% of the 
teachers assigned Number One to "technology" followed by "physical school improvements" and "teacher 
salary increases." The teachers we studied rejected "better salaries = better teaching" as too simple an 
explanation for their professional enactment. It is worth noting that the achievement score gains have 
continued although the planned three year schedule of pay increases has long since been completed, (n 
general, the literature on teacher compensation suggests that salary is important at two points in the career, 
deciding to get in and deciding to get out. But, in between, what teachers teach and how they teach it is not 
influenced by money. It can be influenced by things like standards, testing, supervision, peer influence and 
instructional technology. 

Whether or not plant renovations and teacher salaries are linked to gain scores, there are compelling rea- 
sons like physical safety and the stability of the teaching force to support improvement in those things. 

Beginning in 1995, West Virginia Bell Atlantic connected 700 of the State's 840 school buildings to Bell 
Atlantic's "World School" Internet service. The presence of all ‘other technology related initiatives' in the 
school is related to achievement score gains but accounts for 0.4% of the difference. Those other initiatives 
included computers purchased from other sources, special grant programs and, in many schools, the "World 
School" Internet service. That capability has very probably helped, but 56% of the teachers acknowledged 
that they knew very little about networking and online communications and only a third of the BS/CE 
teachers have classroom access to the Internet. 

During the years in which BS/CE has operated, the State legislature has also required local school councils, 
faculty senates, school-site accreditation visits and a "probation" procedure. Those activities, along with the 
requirement that each school have a "Unified School Improvement Plan," are similar to what other states 
have adopted. The policies are reasonable and even wholesome but they have not been unambiguously 
connected to increases in statewide test scores. There are no states (including West Virginia) In which 
governance modifications have been studied and have been associated with test score gains. 

We have no doubt that changes in school policy. In addition to BS/CE, made some difference. The case of 
"standards" is instructive. The State Department of Education promulgated the "West Virginia Instructional 
Goals and Objectives." In order to win state contracts, both IBM and Jostens Learning correlated their soft- 
ware to that state framework. Once the choice between software vendors had been made, those correlation 

Slate taxes were raised $200 million in 1989 and another $200 million in 1990. 
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matrices became a resource if not a guide for instruction by teachers, And the teachers came to rely on them 
as they integrated BS/CE computers into their teaching, Simultaneously, the State reminded teachers to 
reinforce student basic skills acquisition through the "Teach/Re-Teach" program. The quantitative and the 
qualitative data both support the idea that teachers converged on using BS/CE computers to extend their 
Basic Skills instruction in order to position their students for the related achievement tests. Thus, the 
standards/assessment policies reinforced the policy of technology use Integrated into the curriculum and 
vice versa. 

Our data suggest what expert judgement supports — various policy interventions make various amounts of 
difference and they most probably interact. 

But, we do not have direct evidence about that. In fact, few changes in schooling depend on demonstrated 
improvements in test scores. Text books get adopted, professional development seminars get scheduled, 
whole theories of learning or of school organization get implemented without any evidence that they impact 
test scores. That does not mean that those phenomena are not worthy or that they might not have an effect 
on student achievement, only that those effects — lack of effects — are not taken into account when decisions 
about such initiative are made, BS/CE has what others do not — measured results. 



5.0 CONCLUSIONS 

"We need information to show what works and what doesn't. If we had empirical data, policy- 
makers would be more willing to fund technology and voters would be much more wiling to pay," 

- Lieutenant Governor Kim Robak, Nebraska (Kerry White "A Matter of Policy," Education Week, 
November 10, 1997, v, XVII, n, 1 1, "Technology Counts: Schools and Reform in the Information Age," p 6,) 

The West Virginia legislature funded the Basic Skills/Computer Education program in hopes of levering 
improvements, statewide, in the school achievement of children. On the evidence of the State's change in 
rank relative to other states, and on the evidence of this analysis, the state's expectations were met. 

And West Virginia realized additional outcomes from technology. The schools were able to try out new 
productivity tools, public attitudes toward schools were probably improved. Sixty-two percent of the 
American workforce is already "knowledge workers," people who focus on creating, organizing and com- 
municating information^'^, BS/CE is a major part of positioning the State's children for that future. 

The improvements reported here were powered by successive waves of hardware and software, the earliest 
generation of which reached back to the late 1980s, At BS/CE's inception, 486-megahertz machines were 
still on the horizon and the capital investments necessary to support broadcast-quality audio and video were 
still In the publishers' business plans. If we accept that these gains are a function of that previous generation 
of instructional technology, what will be possible next? 

All jurisdictions have an obligation to reach their own conclusions about what is worth doing. The data sug- 
gest the power of Instructional technology but they also signal aspects of policy strategy and tactics, 

BS/CE is scalable. The expenditure totals are not out of line with other states, (See also Lewis Solmon's 
"Afterword,") The program's components have been well documented by the State, The outcomes are established, 

CEO Forum on Education and Technology, School Technology and Readiness Report: From Pillars to Progress, Washington DC,, October 9, 1997, p 3, The figure 
refers to the 1994 American labor force. 
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Consider the logic of West Virginia's follow-through strategy. Without the yearly concentration of hardware, 
software and training grade-by-grade, no grade would have had a critical mass. A one or two year technolo- 
gy-rich effort would quickly get washed out by children falling back to successive years of technology-poor 
experiences. The performance of the BS/CE lead cohort has distinguished itself and the state. A follow-on 
program called "SUCCESS" is designed to carry the same equipment and procedures through to the upper 
grades. There is reason to believe that as BS/CE helped children, so will "SUCCESS;" and there is no 
reason to believe that there will be less technology in the lives of West Virginia's elementary and secondary 
students and graduates. All of that suggests that maintaining this focus will maintain these positive outcomes. 

As other jurisdictions consider instructional technology as an agent of improvement, is it of interest that a 
defined, affordable package of hardware/software/process innovations can account for a large fraction of the 
test score improvement that is available to public policy intervention? And is it of further interest that, in 
addition to test score gains, those innovations can help position children for a technologically demanding 
economy, society and polity? 

But, BS/CE also has a number of features that are uncommon in the state education policy landscape. The 
features that deliver an installed base critical mass depart from the norm of a small number of computers 
equitably distributed across a large number of classrooms. The choice of software from a fixed set of two 
vendors departs from the conventional ceding of choice among hundreds of vendors to hundreds of schools 
(and often, to thousands of teachers). We believe that part of the explanation for BS/CE's success is the 
defined focus of its implementation. It is worth noting that we did not encounter principals who complained 
that their traditional local autonomy was restricted and neither is there evidence of anything other than impa- 
tience as the upper grades teachers anticipated their turn on the State's hardware implementation schedule. 
» 

Still, policy choices, political choices always honor local values. In American schooling, thousands of 
Jurisdictions make their own choices. It may be that this documentation of the student outcomes associated 
with West Virginia's program of Basic Skills/Computer Education will advance the consideration of similar 
initiatives in otherjurisdictions. 
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I n't raductiian 




The Basic Skills/Computer Education Program (BS/CE) in West Virginia has been one of the most comprehensive 
(in terms of students covered in a state) and long-lived statewide education technology programs ever tried. It is one 
of the few programs that has been in existence long enough to provide answers to the fundamental question of 
whether an infusion of technology tied to the curriculum and associated professional development for teachers affect 
student learning as measured by improvement in scores on tests of basic skills. 

Thus, the Milken Exchange commissioned Interactive, lnc\ to gather information on how the technology was used in 
West Virginia's schools and to determine if technology had any impact on the improvement in the test scores of 
West Virginia's students. This monograph is the report of the results of that study. The principal finding is that 
BS/CE worked. 

Mann, Shakeshaft, Becker, and Kottkamp make this case by pointing out that about 11 percent of the total variance 
(R^= .11 and adjusted R^= .094) in the basic skills achievement gain scores of the 5th-grade students in their sample 
who have had BS/CE since 1991-92 can be explained by a model composed of factors that describe the overall BS/CE 
experience. In other words, about 11 percent of the gain score increase of 5th-grade students can be attributed to 
their participation in BS/CE. Moreover, according to the authors, since about 70 percent of the variation in test scores 
relates to family background, only 30 percent remains that schools can influence. Mann, Shakeshaft, Becker, and 
Kottkamp conclude that the 11 percent of the total variance in the basic skills achievement gain scores (from 1995- 
96 to 1996-97) can be explained by participation in BS/CE which is about one-third of the 30 percent variance 
remaining after taking into account family-related influences. 



Comparing Technolagy wi'tli O'tlier Policy Ini'tia'tives 

These are significant findings and important evidence supporting the claims made by advocates of putting tech- 
nology into all public schools and having it used properly^ But in order to recommend similar but more modern 
policies in the future it is not enough to know that BS/CE is related to test score gains; rather, we must be convinced 
that such policies are at least as effective as others that are of similar cost. Thus, to put Mann, Shakeshaft, Becker, 
and Kottkamp's findings in a context of related research, it is useful to look at effect sizes. These can be derived by 
dividing the increase In test scores associated with the regression coefficient for the intervention by the standard 
deviation of test scores in the sample. The regression coefficients and the standard deviations can be found In Mann, 



' Interactive, Inc. is a firm whose principals, Drs. Dale Mann and Oiarol Shakeshaft are also professors at Teachers College, Columbia University and Hofstra University respectively. 

^ Some people believe that BS/CE is an application of old technology that should not be replicated. They call It "drill and practice" or an instructional learning system. In fact, BS/CE 
required new designs to meet West Virginia's needs, and did meet those needs. Although constructivist teaching and learning have merit in mony situations, a variety of other opplications 
of modern technology moy be appropriate depending upon the need. In studying West Virginia's initiative, it is inevitable that Interactive, Inc. analyzed the effects associated 
with computer-related technology available beginning in the early 1990s. A decade is a long time in computer evolution. Observing that BS/CE made a difference for children is not the 
same thing as endorsing the current or future application of ten-year old hardware and software. The size of the effects documented for West Virginia is interesting especially because of 
the generation of hardware and software studied. Constructivist and higher order thinking skills applications, when added to the drill and practice, may well have an even greater effect. 
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Shakeshaft, Becker, and Kottkamp's analysis^ The effectiveness of an intervention is viewed as the increase in stan- 
dard deviation units of test scores associated with the intervention. This approach permits comparisons of the effects 
of the BS/CE with other attempts to improve instruction such as class size reduction, tutoring, and the like. 

Most effect size calculations begin with regression analyses in which the educational intervention is 
measured by a single variable-either the intervention occurred or did not. The regression coefficient for that "dummy 
variable" divided by the standard deviation of the outcome variable gives us a straightforward measure of the effect 
size. Table 1 provides a summary of some effect size studies developed by Benjamin Bloom (in Educational Researcher, 
June/July, 1984) who cites Walberg (1984). 






Table 1 










Effect Size 


Percentile Equivalent 




Tutorial instruction 


2.00 


98 


D 


Reinforcement 


■ ■ s' ■ W V. 

. . T.20: . .. 




A 


Feedback-corrective (ML) 


1.00 


84 


D 


Cues and explanations 


1.00 




(AID 


Student classroom participation 


1.00 




A 


Student time on task 







A 


Improved reading/study skills 


1.00 




C 


Cooperative learning 


.80 


79 


D 


Homework (graded) 


.80 




D 


Classroom morale 


.60 


73 


A 


Initial cognitive prerequisites 


.60 




C 


Home environment intervention 


• 50‘’ 


69 


D 


Peer and cross-age remedial tutoring 


.40 


66 


D 


Homework (assigned) 


.30 


62 


D 


Higher order questions 


.30 




(DIB 


New science & math curricula 


.3G‘’ 




D 


Teacher expectancy 


.30 




C 


Peer group influence 


.20, 


58 


B 


Advance organizers 


.20 






Socio-economic status 
(for contrast) 


.25 


60 



^ Object of change prod^'smm\ B-lnstructional Material: C-Home environment or peer group; D-Teacher. 

‘‘Averaged or estimated from correlational data or from several effect sizes. 

* Bloom, Benjamin S. “The 2 Sigma Problem: The Search lor Methods of Group Instruction as Effective as One-to-One Tutoring." 
Educational Researcbem-Mi 1984: 6. This table v\ras adapted from Walberg (1984) by Bloom. 



^ The regression coefficients are a somewhat ambiguous mixture of reciprocal influences flowing from putative cause to putative effect in unknown proportions. In short, we can't be sure 
of the direction of the causal influence. Does it always run from what has been arbitrarily designated as "independent variable" to "dependent variable" or is It also sometimes running 
in the reverse direction? For example, if we correlate children's "selfconcept" and "achievement," we get a coefficient that surely reflects both selfconcept raising achievement and achieve- 
ment raising self-concept. This is not simply an abstroct possibility in the Interactive, Inc. analysis; such variables as "Time on computer" and "Teacher attitude" could as well be causes of 
achievement gains as their effects. It is likely that the preponderance of the influence probably does run from putative cause to putative effect, but the other direction must be acknowledged. 
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Levin, Glass, and Meister (1987) provide estimates of annual effect sizes for four educational interventions (Table 2). 
The effects vary by subject matter and by grade level, and of course, by the specific intervention. For example, the 
mean effect size for cross-age tutoring was .79 for a combined peer and adult program, .97 for the peer 
component and .67 for the adult component. The range was 1.02 for second-grade math to .35 for sixth-grade read- 
ing. The mean effect size for reducing class size from 35 to 30 was .06 for math and .03 for reading; reducing class 
size from 35 to 20 produced an effect size of .22 for math and .11 for reading. Adding an additional 30 minutes per 
day to instructional time had a mean effect size of .03 for math and .07 for reading. Finally, a ten-minute daily 
session on a mini-computer for computer-assisted instruction had a mean size effect of .12 for math and .23 for read- 
ing. Each standard deviation is approximately equal to gains of an academic year of 10 months, so each tenth of 
a standard deviation can be viewed as about 1 month of achievement gain per year of Intervention (Levin et al, 1987). 



Table 






V.. 4 

i ' : I 

^ 



Intervention 


ivtatnematics 


Reading 


Mean 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Mean 


Grade 2 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


Computer-Assisted 
Instruction [10-minute dai 
session on mini-computer) 


y 
























Overall 


.12" 


.13" 






is" 




.23" 








.23" 






Computation 


.30" 






£i 

in 

OJ 




Vocabulary 


.25" 






in 

OJ 






Concepts 








b 

o 

cr 




Compre- 

hension 


.20" 






.20" 






Application 


.10" 






.lO" 
















Cross-Age Tutoring 

(Boise model) 


























Combined peer and 
adult program 


.79" 


1.02 


.91 


.79 


.68 


.55 


.42" 


.50 


.46 


.42 


.39 


.35 


1-Peer Component 


.97" 


1.02 


.91 








.48" 


.50 


.46 








1 -Adult Component 


.e?" 






.79 


.68 


.55 


.sa" 






.42 


.39 


.35 


Reducing Class Size 


























From 35 to 30 


.06 












.03 












From 30 to 25 


.07 












.04 












From 25 to 20 


.09 












.05 










/ 


From 35-20 


.22 












.11 












Increasing Instructional 
Time (additional 30 minute 
per day for each subject) 


s 

.03 


.02 






.04 




.07 


.08 






.07 





Note: average for grades 2 and 3; average for grades 4. 5, and 6; average for grades 2 through 6. 



levin, Henry M., Glass, G., Meister, Gail R. “Cost-Effectiveness of Computer-Assisted Instruction.” Evaluation Review^Ql 11 No. 1, February 1987 p.57. 
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The Mann, Shakeshaft, Becker, and Kottkamp study differs from the ones referred to above in that the BS/CE inter- 
vention is measured not by a single independent variable, but by a set of nine components of that program (grouped 
into three categories), some derived from a factor analysis of responses to several survey questions. In Table 3 we 
divided the regression coefficients on each factor representation by the standard deviation of the test score outcome 
variable to get an “effect size." The quotation marks are used because these effect sizes are unusual ones— not the 
effect of an intervention as compared to no intervention, but rather the effect of more of the factor versus less. 
Moreover, regression coefficients' size depends on units of the factor, but we do not know what the units of each 
factor are. As an alternative, we provide the possible ranges of each variable as it is constructed to help with the inter- 
pretation. Assuming that the factor analysis converts each factor into comparable units, the effect size measures tell 
us the effect of having more versus less and allow us to compare effects of various aspects of the program. Most of 
the independent variables are scales within a relatively narrow range where a higher score means more of that 
attitude or input. 



I Table 3 




>r.L^ i ' 




Range of 
Variable 


Regression 

Coefficient 


Standard 
Oeuiation of 
Outcome 
Variable 


Effect 

Size 


Standardized 

Beta 

Coefficient 


Significance 


Hardware and Software 
Access and Use 


Connputer and BS/CE use: 
time spent on BS/CE and 
time spent using other software 


3-15 


1 

1.453 


16.240 


0.089 


0.240 


0.000 


Software access: choice of vendor; 
use of BS/CE specific software 


1-2 


5.958 


16.240 


0.367 


0.184 


0.000 


Computer access: 

availability of computers to students 


1-2 


2.126 


16.240 


0.131 


0.068 


0.115 


Other technology access: 
software use other than BS/CE 


0-4 


0.715 


16.240 


0.044 


0.051 


0.278 


Attitudes 


Student attitudes 
towards computers 


2-4 ' 


2.742 


16.240 


0.169 


0.099 


0.025 


Teacher attitudes ! 

towards computers: teacher willingness to : 

continue learning about computers 
and technology 


2-10 


1 .069 ' 


16.240 


0.066 


0.123 


0.014 


Teacher confidence: 

teacher comfort level and 

confidence in using computers i 


8-40 


i 

0.310 


16.240 


0.019 


0.030 


0.596 


Teacher Training and Involvement 


Involvement: in the planning 
and implementation of BS/CE 


5-1 


-1.959 


16.240 


- 0.121 


- 0.097 


0.065 


Training: amount of time teachers 

spent learning on the computer on their own 


1-5 


1.109 


16.240 


0.068 


0.095 


0.091 
















Sum of all effect sizes 








1.074 
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The input that can be quantified in the most meaningful way is student time^ The variable used combines teacher 
and student responses to questions on how many days per week and how many minutes per day the students work 
with computers at school (both time spent on BS/CE and time spent using other software). The result is a rank-order 
scale from 3 (1 5 minutes or less, one day per week in math, reading, and writing) to 1 5 (one hour or more, five days 
a week in the same three subjects). This 5th-grade computer use factor alone yields an effect size of .089. 

The "basic skills/software access" factor (2 if Jostens, 1 if IBM) hasan effect size of .367^and the "student attitudes" 
factor (student attitudes about importance of learning about computers compared to learning to read and write and 
compared to learning other subjects, range from 2 to 4) has an effect size of .1 69. Both of these are smaller than or 
equivalent to (depending on grade level and discipline) the effect sizes of cross-age tutoring cited in Levin et al, but 
are equal to or larger than the sizes of effects of computer-assisted instruction, class size reduction or increasing 
instructional time. 

However, if we are really interested in the effects of the total BS/CE program, we should look at the sum of effect sizes 
of all its components. The point is not which individual components of the model were most effective, but rather that 
all the components of the model made a difference together. Assuming that there is no colinearity among the nine 
factors, there is some logic in simply summing the size of each of the effects^ When we do that the total effect 
size is 1.074^ The sum of the Mann, Shakeshaft, Becker, and Kottkamp effect sizes from these individual, unaggregated 
components of instructional technology are larger than those in Levin et al's 1987 report. Given the technological/ 
pedagogical advances since then, the gain from more recent technology is what would be expected, just as, when even 
more recent hardware, software and teaching practice is assessed, the effect sizes can be predicted to be larger. 

The largest individual effect (.367) comes from software access. As noted above, the total effect of all significant 
factors is 1.074. If all factors were perfectly correlated, the overall effect would be .367. If there were no colinearity, 
the overall effect would be 1.074. The reality is probably somewhere between these two numbers. Given that Levin 
et al found mean effect sizes for computer assisted instruction of .12 for math and .23 for reading, the effect of 
BS/CE is perhaps at least two to five times as large. This is not surprising given that the CAI in the Levin et al study 
involved only a ten-minute session each day, and occurred at a much earlier stage of development of educational 
hardware and software. 

Kulik (in Baker Et O'Neil, 1994) looked at about a dozen meta analyses of studies of effectiveness of computer-based 
instruction. The estimates of the magnitudes of the effects ranged from .22 at the low end (18 studies conducted in 
elementary and secondary science courses) to .57 (18 studies in special education classes). The mean effect size was 
to raise test scores by .35 standard deviations, or from the 50th to the 64th percentile. As noted earlier. Levin et al 
found mean effect sizes of .12 and .23 for math and reading, respectively, for grades 2 through 6. 

The four factors found by Mann, Shakeshaft, Becker, and Kottkamp to have statistically significant positive regres- 
sion coefficients®, and hence, relevant effect sizes were "software access," "time students spend with computers," 
student and teacher attitudes, and principal leadership. The first two are probably most comparable to other 
studies. Thus, this combined effect size of materials and time of .456 falls within the range of the Kulik survey and 
is higher than the effect of ten minutes per day of CAI in the Levin et al study. The .456 effect size says use of 
materials for more time led to an increase in test scores from the 50th to about the 67th percentile, or a 4.6-month 
achievement gain. This conclusion depends upon the metric on which each independent variable is measured. 

* Obviously, other factors can be quantified, however they may not be standard measures of equal units. 

^ There was BS/CE software and when it was used, it made a difference. 

‘ There appears to be no significant correlations among the 9 independent variables. The largest is .38 between the measure of teacher control and a measure of teacher attitudes toward 
using computers and the BS/CE initiative. Thus the betas can be considered as additive. 

^ The coefficient on teacher leadership is negative because teacher leadership is inferred from a question about leadership of the principal. Since a higher score indicates more leadership 
from the principal, it means less from the teacher. Hence, this negative coefficient should be included with Its sign reversed. 

‘ Significance at the .02 level 
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Mann, Shakeshaft, Becker, and Kottkamp’s result gets us to an effect size of 1.074 when we add the effects 
of improved student and teacher attitudes about technology and teacher training and involvement— which are 
engendered by BS/CE independent of tht amount of time spent using the program’s materials. If these really are inde- 
pendent components of BS/CE, then the program moved students from the 50th to about the 85th percentile in a 
year, or achieved a gain of 10.7 months over and above that which would have been expected from typical classroom 
instruction not aided by BS/CE. 

The authors’ conclusion that BS/CE had a positive effect on West Virginia appears to hold up under the scrutiny of 
effect size analysis. The results from West Virginia are stronger than studies of effect sizes of CAI; however, BS/CE was 
a project of much larger scale than most of the innovations considered elsewhere. Technology has been shown to have 
a positive effect. It will be left to the reader to assess how big is a move from the 50th to the 67th or 85th percentile 
(given our interpretation of the metric of the independent variables). 



Comparing t:he Caste of InsteucCional 
Technolagy te Ittlier Interventions 

Finally, it is important to examine the improvement per dollar spent on BS/CE compared to actual or potential gains 
per dollar spent on other interventions. BS/CE cost $7 million per year to add technology and provide teacher train- 
ing to one grade level across the stated For comparison purposes, let us look at the hypothetical cost of reducing class 
size in West Virginia from the current level of 21 students per class to 15 students per class (Table 4). There are cur- 
rently 301,314 students in K-12 and 14,348 classes and teachers (assuming one teacher per class). To reduce class size 
to 15 would require 5,739 additional classes and teachers. At the current average teacher salary of $33,396, the total 
cost of the class size reduction program would be $191,670,140 for additional teacher salaries alone. This does not 
include cost of adding physical classrooms, which has turned out to be a major problem for California’s class size 
reduction effort. 



Table 4 



Cdi^ of CiasSrSize Reduction 






K-12 enrollment (1997-98)* 


301,314 


Average teacher salary (1997-98)* 


$ 33.396 


Class size (1996-97)* 


21 


Number of classes 


14,348 


Number of classes if class size = 15 


20,088 


Additional classes & teachers needed 


5,739 


Cost of new teachers 


$ 191,670,140 


Cost per grade 


$ 14,743,857 


Cost in yr 2 


$ 29,489,475 


Cost in yr 3 


$ 44,235,093 


Cost in yr 4 


$ 58,980,711 


Cost in yr 5 . 


$ 73,726,329 


Cost in yr 6 


$ 88,471,947 


Cost in yr 7 


$ 103,217,565 


* Source: West Virginia Department of Education 



^ West Virginia Department of Education, Private Memorandum from State Superintendent of Schools, Henry R. Marockie, November 19, 1998. 
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If class size were reduced in one grade each year (to make such a program similar to the technology program), the cost 
for each additional grade would be $14,743,857. This is double the cost per grade of technology in the form of BS/CE. 
Moreover, not only will the cost of the new grade be incurred each year; but also, a similar cost will arise for each of 
the grades whose class sizes were reduced in previous years. That is, for class size reduction, the salary cost of the 
additional teachers comes up every year. The major costs of the technology program are incurred once, in the year 
the program is introduced into a grade’°. 



By the time a student has completed the 5th grade. West Virginia will have spent $42 million on BS/CE or $7 million 
each year on the grade (K-5) that the student has entered. A class size reduction effort over the same period of time 
would have cost $310 million because as additional teachers are added to an additional grade level each year, the 
salaries of teachers added to lower grades in previous years still must be paid. A one-time expenditure of $7 million to 
put technology into all classrooms of a particular grade benefits children who enter that grade year after year at 
little additional cost (except maintenance, updating, etc.). When calculating the cost of each program per student, we 
must take into account that by the time the 5th-grade students reach the 5th grade, five other cohorts of kinder- 
gartners will have benefited from technology or reduced class size in kindergarten, four additional cohorts will have 
benefited from changes in 1st grade and so on. Thus the total cost figures presented above translate into $636 per 
student per year for class size reduction and $86 for BS/CE (Table 5). 



Table 


•5 


rs 


.::r* ''^brhp 


larison of^:l 


Edi^s:ani::l^ 


Benelltsac 


ifrReduG 














One Year 
Effect Sizes 




Effect size of 
reducing 
class size 
from 35 to 30 
on math 
scores 


Effect size of 
reducing 
class size 
from 35 to 
20 on math 
scores 


Effect size of 
reducing 
class size 
from 21 to 
15 on math 
scores 


Effect 
size of 
“materials” 
aspect of 
technology 
program 


Effect size 
of time 
students 
spend with 
technology 


Effect 
size of 
“materials” 
aspect of 
technology 
program 


Effect size 
of time 
students 
spend with 
technology 






0.06 


0.22 


0.096 


0.367 


0.089 


0.367 


0.089 


E.S. PER 5 ST 
DECR FROM 30-20 






0.08 












Months gain 








0.96 


3.67 


0.89 


3.67 


0.89 










Cost in 
each year 


Cost $7m 
per year 


Cost $7m 
per year 


Cast $7m per 
year plus 25% 
maintenance 


Cast $7m per 
year plus 25% 
maintenance 


Spending yr 1 








$14,743,857 


$7,000,000 


$7,000,000 


$7,000,00C 


$7,OOO,O0C' 


Spending yr 2 








$29,487,714 


$7,000,000 


$7,000,000 


$8,750,00C 


) $8,750,00) 


Spending yr 3 








$44,231,571 


$7,000,000 


$7,000,000 


$10,500,000 


$10,500,000 


Spending yr 4 








$58,975,428 


$7,000,000 


$7,000,000 


$12,250,00C 


) $12,250,0ai 


Spending yr 5 








$73,719,285 


$7,000,000 


$7,000,000 


$14,000,00C 


1 $14,000,000 


Spending yr 6 








$88,463,142 


$7,000,000 


$7,000,000 


$15,750,0a 


1 $15,750,00C 


Amount spent by 
the time a student 
finishes 5th grade 


' ^ 






®9, 620,99: 


’ $42,000,000 


$42,000,000 


$68,250,00C 


) $68,250,00) 


Students 




301,314 C 


lum students 












St per grade 


K 


23.178 


23.342 














1 


23,178 


46.356 














2 


23.178 


69,534 














3 


23,178 


92,712 














4 


23.178 


115,890 














5 


23,178 


139,068 1 


















486,902 












Amount spent 
per St per yr 








$635.90 


$86.26 


$86.26 


$140.17 


$140.17 


$ per 1 mo gain 
in math scores 
in 5th grade 








$662.40 


$23.50 


$96.92 


$38.19 


$157.50 



Later we show the impact of annual maintenance and upgrading costs of technology. 
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These per student costs now have to be related to test score gains. Levin et al estimate the effect size of reducing 
class size from 35 to 30 as .06 and from 35 to 20 as .22. Taking account of the apparent greater effect (per reduc- 
tion of 5 students) when class size is reduced from 30 to 20 than when the reduction is from 35 to 30, and 
recognizing that the reduction suggested for West Virginia is six students, we estimate roughly that, based on the 
Levin et al study, the effect size of class size reduction in West Virginia would be .096. This compares with effect sizes 
of .367 for the materials aspect of technology and .089 for time students spend with technology. Each .1 effect size 
reflects a one-month gain in grade level due to the intervention. Thus, a one-month gain from reducing class 
size from 21 to 15 would cost $662 and a similar improvement would cost $97 using BS/CE if we consider only the 
effect of time spent with technology by students. The effect of materials implies that a one-month gain costs $23.50". 

So far, we have assumed a one-time cost for each additional year of BS/CE. Usually a technology program requires 
annual expenditures for maintenance, upgrading of hardware and software and ongoing teacher training. Thus we 
recalculated the costs of BS/CE to include an expenditure of 25% of implementation costs in each year after 
implementation (Table 5). This cost increase raises the cost of obtaining a one month gain in test scores in the 5th 
grade from $23.50 to $38.19 for software access and from $96 to $157.50 for time students spend with technology. 
These costs are still several times less than the costs of achieving the same test score growth by class size reduction. 

These estimates are rather crude, and are based upon a number of assumptions. The differences between the two 
reforms we compared are very large. Thus, if our assumptions are too favorable to the intervention with technology, 
reasonable but less favorable assumptions are likely still to show greater cost-effectiveness for technology than 
for class size reduction. However, they lead us to conclude that not only is there a statistically significant relation- 
ship between BS/CE and test score gains; and not only can these gains be translated into effect sizes comparable to 
those of other interventions, but also, the gains from programs that update BS/CE's positive features can be achieved 
at a much lower cost than could similar gains from a currently very popular alternative intervention, namely, class 
size reduction. 

Lewis C. Solmon 
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We are using Levin's figures on math scores whereas in West Virginia we are looking at a test of basic skills that is a specific Stanford/West Virginia product and includes math, reading, 
and language arts. Implicitly we are assuming that each area can be improved equally with a particular treatment. 
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